CLOUDP-295785 - staging support for image building #336

MaciejKaras · 2025-08-11T19:00:03Z

Summary

PR integrates staging build scenario with atomic_pipeline.py and the e2e tests. staging repositories are created in AWS ECR under /staging dir. List of new staging/ repositories:

Example AWS page with details:

Additional changes:

re-added missing latest tag for staging builds. This is used for local testing
renamed mongodb-enterprise-ops-manager repository to mongodb-enterprise-ops-manager-ubi (which is used in e2e tests and production

Proof of Work

Building all images is passing -> https://spruce.mongodb.com/version/689a0407e7099d000753a536/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC

Example image stored together with signature (latest tag is mutable)-> https://us-east-1.console.aws.amazon.com/ecr/repositories/private/268558157000/staging/mongodb-kubernetes-init-database?region=us-east-1

Hacked the build_scenario.py and evg-private-context to force staging scenario and make e2e test use the staging paths. Patch with all tests succeeding -> https://spruce.mongodb.com/version/689c67f0ccbf5100079e1e00/tasks?sorts=STATUS%3AASC%3BBASE_STATUS%3ADESC

Checklist

Have you linked a jira ticket and/or is the ticket in the title?
Have you checked whether your jira ticket required DOCSP changes?
Have you added changelog file?
- use skip-changelog label if not needed
- refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

Fix build scenario Remove create and push manifests Continue improvement to main Simplify main and build_context missed Pass Build Configuration object directly Use legacy and new pipeline Fix Remove --include Rename MCO test image Multi platform builds, with buildx TODOs Implement is_release_step_executed() Fix init appdb image Import sort black formatting Some cleaning and version adjustments Adapt main to new build config Add buildscenario to buildconfig Handle build env Renaming, usage of high level config All images build pass on EVG Lint Explicit image type, support custom build_path Replace old by new pipeline in EVG Add documentation Split in multiple files, cleanup WIP, passing builds on staging temp + multi arch manifests Replace usage of sonar Remove namespace Remove pin_at and build_id Copied pipeline, removed daily builds and --exclude

This reverts commit 426e522.

…staging

# Conflicts: # build_info.json # scripts/release/atomic_pipeline.py # scripts/release/build/build_info.py # scripts/release/build/build_info_test.py # scripts/release/build/build_scenario.py # scripts/release/build/image_build_configuration.py # scripts/release/build/image_build_process.py # scripts/release/pipeline_main.py

# Conflicts: # scripts/release/atomic_pipeline.py # scripts/release/build/build_info.py # scripts/release/build/build_info_test.py

MaciejKaras · 2025-08-13T12:29:17Z

scripts/dev/contexts/variables/om60

@@ -13,10 +13,10 @@ export CUSTOM_OM_VERSION
 export CUSTOM_MDB_VERSION=6.0.21
 export CUSTOM_MDB_PREV_VERSION=5.0.7

-export AGENT_VERSION=12.0.33.7866-1
+export AGENT_VERSION=12.0.34.7888-1


this had to change, because the previously used agent version does not appear in release.json and there is no way to rebuild it.

MaciejKaras · 2025-08-13T12:48:30Z

scripts/dev/contexts/local-defaults-context

we should later change BASE_REPO_URL_SHARED to use 268558157000.dkr.ecr.us-east-1.amazonaws.com/staging when we have staging images already

q: When would this be and how would we trigger the change then?

q: Are we going to use staging registry the same way we were using ECR /dev before ?
Does it mean we will push to staging with both tags: the commit sha and latest ?

(just for my understanding, but I think it makes sense)

As I understand, please correct me if I'm wrong, latest images from /dev were used for local testing, so we can reuse most recent images. Previously the latest tag was added when the image was built for master commits. To preserve existing behaviour we are going to add latest tag also when we merge to master (staging build), but those images will land now in /staging repositories.

Because of this we will need to change our scripts/dev/contexts/local-defaults-context to use \staging base repository instead.

Sounds good to me ! Thanks for clarifying :)

(I'll let @mircea-cosbuc resolve the thread as he commented as well)

build_info.json

mircea-cosbuc · 2025-08-13T13:53:45Z

scripts/dev/contexts/local-defaults-context

q: When would this be and how would we trigger the change then?

mircea-cosbuc · 2025-08-13T13:55:08Z

scripts/release/build/build_scenario.py

@@ -28,7 +28,7 @@ def infer_scenario_from_environment(cls) -> "BuildScenario":
            scenario = BuildScenario.RELEASE
            logger.info(f"Build scenario: {scenario} (git_tag: {git_tag})")
        elif is_patch or is_evg:
-            scenario = BuildScenario.PATCH
+            scenario = BuildScenario.STAGING


q: The PR is ready for review. Are those comments meant to be addressed before merging?

build_info.json

Julien-Ben · 2025-08-13T14:11:14Z

scripts/release/atomic_pipeline.py

-        f"Building {image_full_uri} for platforms={build_configuration.platforms}, dockerfile args: {build_args}"
-    )
+    tags = [f"{build_configuration.registry}:{build_configuration.version}"]
+    if build_configuration.latest_tag:


Is it possible to handle tags as a list in the build configuration instead of treating latest as a special case ?

for latests yes, but not for the olm tag where we will have a dynamic timestamp suffix. Because of this I've chose the bool rather than array

scripts/release/build/build_scenario.py

scripts/release/release_info_test.py

nammn · 2025-08-13T13:08:13Z

scripts/release/build/build_scenario.py

@@ -28,7 +28,7 @@ def infer_scenario_from_environment(cls) -> "BuildScenario":
            scenario = BuildScenario.RELEASE
            logger.info(f"Build scenario: {scenario} (git_tag: {git_tag})")
        elif is_patch or is_evg:
-            scenario = BuildScenario.PATCH
+            scenario = BuildScenario.STAGING


when building something new we should keep testability in mind - I think if we need to hardcode this - this means we forgot something as part of the planning

agree! Reverted the change and we can think how to test this properly

nammn · 2025-08-13T13:09:16Z

scripts/dev/contexts/evg-private-context

+STAGING_REPO_URL="268558157000.dkr.ecr.us-east-1.amazonaws.com/staging"
+COMMIT_SHA_SHORT=$(git rev-parse --short=8 HEAD)
+
+if [ "${is_patch:-false}" = "true" ]; then


we should have a way to test this change without doing these hacks to verify it. Otherwise, we will run into the same problem like the last pipeline

agree! Reverted the change and we can think how to test this properly

build_info.json

# Conflicts: # scripts/release/tests/build_info_test.py

Julien-Ben added 30 commits July 29, 2025 17:30

Remove file

675bee4

Put lib back in dependencies

833e25f

add todo

15e7f51

Fix

120c1af

Remove multi arch call, fix test image path

c9ceabf

Fix agent version for default case

fb87f4d

Lindt

c05e180

isort

747c4ba

Cleanup TODOs

03fd9b8

Rename arch -> platform

1fbb8d5

Don't rely on exception to check for builder existence

e9a524f

Remove unused variables

fa6b899

Pre commit

426e522

Cleanup

6890858

Correct build envs

aab9592

Lindt

33173bb

Update Makefile

74e867c

Add TODO

b13b054

Revert "Pre commit"

832ce61

This reverts commit 426e522.

Remove agent unused functions

bc23827

Remove debug

03062c3

Remove args iteration for multi platform

4d4e4e9

Cleanup unused stuff

7c9967a

Cleanup

a7c63c9

Rename file

742e784

Remove cli sbom

1f0a21b

Renamed image building file

813d539

Freeze python on whales

c06061b

Lint

5f9d49a

MaciejKaras added 2 commits August 12, 2025 12:33

Fix unit tests

8fa0e98

Merge branch 'maciejk/use-ar-related-versions' into maciejk/ar-image-…

c4f4999

…staging

Base automatically changed from maciejk/use-ar-related-versions to master August 12, 2025 10:58

MaciejKaras added 7 commits August 12, 2025 15:36

remove duplicate entries

b1fabb4

Fix unit tests

91f7054

Merge branch 'master' into maciejk/ar-image-staging

5b9a006

Update mongodb-enterprise-ops-manager-ubi image name

907e1c4

Update OM6 agent version (previous was deleted)

2a0438c

Merge branch 'master' into maciejk/ar-image-staging

ee7d8ad

# Conflicts: # scripts/release/atomic_pipeline.py # scripts/release/build/build_info.py # scripts/release/build/build_info_test.py

MaciejKaras commented Aug 13, 2025

View reviewed changes

MaciejKaras marked this pull request as ready for review August 13, 2025 12:55

MaciejKaras requested a review from a team as a code owner August 13, 2025 12:55

MaciejKaras requested review from m1kola, lucian-tosa, mircea-cosbuc, nammn, Julien-Ben and viveksinghggits and removed request for m1kola and lucian-tosa August 13, 2025 12:55

mircea-cosbuc reviewed Aug 13, 2025

View reviewed changes

Julien-Ben reviewed Aug 13, 2025

View reviewed changes

Remove TODO for STAGING build scenario

c8945e9

MaciejKaras commented Aug 13, 2025

View reviewed changes

scripts/release/build/build_scenario.py Show resolved Hide resolved

Add commit_sha in logger

51a554d

nammn reviewed Aug 13, 2025

View reviewed changes

MaciejKaras added 2 commits August 14, 2025 14:09

Remove STAGING overrides

92bbd95

Merge branch 'master' into maciejk/ar-image-staging

18582c7

# Conflicts: # scripts/release/tests/build_info_test.py

CLOUDP-295785 - staging support for image building #336

Are you sure you want to change the base?

CLOUDP-295785 - staging support for image building #336

Uh oh!

Conversation

MaciejKaras commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of Work

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaciejKaras Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaciejKaras Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MaciejKaras commented Aug 11, 2025 •

edited

Loading

MaciejKaras Aug 13, 2025 •

edited

Loading

MaciejKaras Aug 13, 2025 •

edited

Loading